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We search for the production of single top quarks via flavor-changing neutral current couplings 
of a gluon to the top quark and a charm (c) or up (m) quark. We analyze 230 pb~^ of lepton+jets 
data from pp coUisions at a center of mass energy of 1.96 ToV collected by the DO detector at the 
Fermilab Tevatron Collider. We observe no significant deviation from standard model predictions, 
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and hence set upper limits on the anomalous coupling parameters Kg/A and k^/A, where Kg define 
the strength of teg and tug couplings, and A defines the scale of new physics. The limits at 95% 
C.L. are: k^/A < 0.15 TeV"^ and k^/A < 0.037 TeV'V 

PACS numbers: 11.30.Hv; 13.85.Rm; 14.65.Ha; 14.70.Dj 



Top quarks were discovered in 1995 by the CDF and 
DO collaborations [l| at the Fermilab Tevatron Collider 
in tt pair production involving strong interactions. The 
standard model (SM) also predicts the production of sin- 
gle top quarks via electroweak exchange of a boson 
with cross sections of 0.88 pb in the s-channel {tb) and 
1.98 pb in the t-channel (tqb) [2]. At the 95% C.L., limits 
set by DO are 6.4 pb on the s-channel cross section and 
5.0 pb on the ^-channel cross section [1], and those set 
by CDF are 13.6 pb and 10.1 pb, respectively [1]. DO re- 
cently reported evidence for the production of single top 
quarks at significance of 3.4 standard deviations [5|. 

Since the top quark's discovery, several precision mea- 
surements have been made of its properties. Its large 
mass close to the electroweak symmetry-breaking scale 
suggests that any anomalous coupling could possibly be 
first observed in the top quark sector. One form of 
anomalous couplings can give rise to a single top quark 
in the final state through flavor-changing neutral cur- 
rent (FCNC) interactions with a charm or an up quark, 
involvingthe exchange of a photon, a Z boson, or a 
gluon [a, |7| ■ Although such interactions can be produced 
by higher-order radiative corrections in the SM, the effect 
is too small to be observed Q . Any observable signal in- 
dicating the presence of such couplings would be evidence 
of physics beyond the SM and would shed additional light 
on flavor physics in the top quark sector. 

At present, strong constraints exist for FCNC pro- 
cesses via a photon or a Z boson exchange [2, [13, HI 
from studies of both the production and decay of top 
quarks. In this Letter, we present a search for the pro- 
duction of single top quarks via FCNC couplings of a 
gluon to the top quark in data collected from pp colli- 
sions at y/s — 1.96 TeV using the DO detector. This is 
the flrst search of its kind at hadron colliders. We con- 
sider top quark production rather than decay, since the 
former is more sensitive to the anomalous couplings (ng) 
involving the gluon [13] ■ To date, the best constraints on 
these processes are from the DESY ep Collider (HERA): 
Kg /A < 0.4 TeV-\ at 95% C.L. flS\, where A is the new 
physics cut-off scale. 

We consider events where the top quark decays into a 
b quark and a W boson, and the latter subsequently de- 
cays leptonically (W iv, where £ = e, or r, with the 
T decaying to either an electron or a muon, and two neu- 
trinos). This gives rise to an event with a charged lepton 
of high transverse momentum (pr), significant missing 
transverse energy {^t) from the neutrinos, and at least 
two jets, one that is a 6-quark jet (from the top quark 
decay), and the other from a c quark, u quark, or a gluon. 



Displaced secondary vertices are used to identify b jets [3| . 
The largest physics backgrounds to these events are from 
SM production of M^-f jets and tt, along with smaller con- 
tributions from SM production of single top quarks (tb 
and tqb) and dibosons {WW and WZ). An additional 
source of background is from multijet events in which a 
jet is incorrectly identified as an electron or in which a 
muon from a heavy flavor decay appears isolated. 

The DO detector is described elsewhere [l3|- We use 
the same dataset, basic event selections and background 
modeling as in our SM single top quark search [Sj]; how- 
ever, since the FCNC signal processes have only one b 
quark in the flnal state, we consider here events with 
only one 6-tagged jet. In addition, we include here the 
SM single top quark processes {tb and tqb) in the back- 
ground model. The data were recorded between August 
2002 and March 2004 with a total integrated luminosity 
of 230 ± 15 pb~^ [l3| and were collected using a trigger 
that required a reconstructed jet and an electromagnetic 
energy cluster in the electron channel, or a jet and a 
muon in the muon channel. 

We model the FCNC signal kinematics using a parton- 
level leading order (LO) matrix element event generator 
CompHEP [i3|. All vertices involving the top quark, 
a charm or an up quark, and the gluon are taken into 
account. Representative 2—^2 Feynman diagrams are 
shown in Fig. [TJ Decays of the top quark and W bo- 
son are done in CompHEP to take into account all spin- 
dependent effects. The effects of FCNC couplings are 
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FIG. 1: Representative 2^2 Feynman diagrams for single top 
quark production through flavor-changing neutral currents in- 
volving the gluon. 

parameterized in a model-independent way via an effec- 
tive Lagrangian [12] that is a linear function of the factor 
Kg/A. The production cross section of single top quarks 
thus depends quadratically on Kg/ A, and for certain val- 
ues of Kg /A can be significantly larger than that in the 
SM, as shown in TablcUl The cross sections are evaluated 
at a top quark mass of nit — 175 GeV, with the factoriza- 



tion and renormalization scales set to Q 



it The LO 



cross sections are scaled to next-to-leading (NLO) order 
by a if-factor (NLO/LO cross section ratio) of 1.6 [l^ . 
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TABLE L The production cross sections of single top quarks 
through a gluon exchange in pp collisions at ^/s = 1.96 TeV 
for different values of Kg/ A, as obtained from CompHEP and 
scaled to NLO by a i('-factor of 1.6. 



TABLE II: Summary of event selections. 



kJA [TeV-^] 


a 


(t) [pb] 




teg 


tug 




« = 


0) {K'g = 0) 


0.01 


0.05 


0.88 


0.03 


0.45 


7.92 


0.07 


2.40 


42.61 


0.11 


5.86 


104.78 



The effect of FCNC couplings on the top quark decay 
is negligible for Kg/A <, 0.2 TeV~^ [12] ■ In this range of 
Kg/ A, it is therefore safe to assume that the top quark 
decays into a W boson and a b quark with a branching 
fraction close to unity, as in the standard model, and 
hence, the cross section a(t) multiplied by the branch- 
ing fraction for the process t Wb ivb would also 
depend quadratically on Kg/ A. We may therefore model 
the shapes of the signal kinematic variables at any one 
value of Kg/ A and scale the distributions appropriately 
to obtain them at any other value. We choose that value 
of Kg/ A to be 0.03 TeV"^ in CompHEP and generate 
two sets of signal events: one for the teg process only, in 
which Kg is set to zero, and the other for the tug process 
only, in which Kg is set to zero. 

The parton-level s amp les from CompHEP are pro- 
cessed with PYTHIA [l8[ for fragmentation, hadroniza- 
tion, and modeling of the underlying event, using the 
CTEQSl 119i] parton distribution functions. We use 



TAUOLA [20| for the tau lepton decays and evtgen [21 | 
for the 6-hadron decays. The generated events are pro- 
cessed through a GEANT-based [i^l simulation of the 
DO detector, and normalized to the NLO cross sections 
for Kg /A = 0.03 TeV~^ For the backgrounds, the Monte 
Carlo (MC) simulated samples are generated and nor- 
malized as described in Ref. Q . 

The event selections [3^] applied to the simulated signals 
and backgrounds and to the DO data are summarized in 
Table HIl The resulting numbers of events from all sam- 
ples, along with their systematic uncertainties described 
later, are shown in Table Hill We find that the observed 
numbers of events agree with the predicted numbers for 
the SM backgrounds within uncertainties in both the 
electron and muon channels, and that the FCNC signals 
are a tiny fraction. We therefore construct multivari- 
ate discriminants using neural networks to separate the 
expected signal from the background and enhance the 
sensitivity. 

We use MLPfit implementation [2^ of neural networks 
with ten input variables representing individual object 
kinematics, global event kinematics, and angular corre- 
lations. These are listed in Table IIVI with the distri- 





Electron channel Muon channel 


Lepton 


Et > 15 GeV PT > 15 GeV 
\v\ < 1-1 I'?! < 2.0 




15 < < 200 GeV 


Jets 


2, 3 or 4 jets, Et > 15 GeV, Ir;] < 3.4 
SrOetl) > 25 GeV, |77(jetl)i < 2.5 
exactly one 6-tagged jet 



TABLE III: Event yields after all selections for the electron 
and muon channels. The signal yields are evaluated at Kg /A = 
0.03 TeV~^. The yields for tt include both lepton-|-jets and 
dilepton final states, and those from M^-|-jets also include the 
diboson backgrounds. 



Source 


Electron channel Muon channel 


teg 


0.6 ±0.2 


0.6 ±0.2 


tug 


8.4 ±2.1 


9.8 ±2.7 


SM single top {tb+tqb) 


6.4 ±1.4 


6.1 ± 1.4 


ti 


31.8 ±6.9 


31.4 ±7.0 


W+jets 


84.6 ±10.2 


76.8 ±8.5 


Multijets 


13.7 ±4.3 


17.2 ±1.5 


Total SM background 


136.5 ±13.4 


131.5 ±12.7 


Observed no. of events 


134 


118 



bution of one representative variable shown in Fig. [5^. 



The combination of several variables in this manner al- 
lows us to separate the FCNC signals not only from the 
dominant backgrounds (W^±jets and tt) but also from 
the SM single top quark processes as can be seen in 
Fig. [^b where the neural network outputs for the com- 
bined electron and muon channels are shown from differ- 
ent sources normalized to unity. Here the FCNC signal is 
for the summed teg and tug processes, each evaluated at 
Kg/ A = 0.03 TcV"^ Fig.H; shows the output distribu- 
tions normalized to DO data with backgrounds summed. 
Since the observed spectrum agrees with the predicted 
SM background, we set upper limits on the FCNC cou- 
pling parameters Kg /A and Kg/ A. 

To estimate systematic uncertainties, we consider ef- 
fects that alter the overall normalization of the distri- 
butions and those that also change their shapes. The 
dominant normalization effects are from lepton identifi- 
cation (4%), integrated luminosity measurement (6.5%), 
and cross section estimates. The uncertainties on cross 
sections vary from 9% for diboson production to 16% 
for SM single top quark production and 18% for ti sam- 
ples [2^. The latter two include the uncertainty due to 
the top quark mass. For the FCNC signal, we factor out 
the parameter (k^/A)^ from the cross section, and as- 
sume an uncertainty of 15% on the remaining quantity 
based on a discussion in Ref. [17| on how the theoreti- 
cal predictions depend on the particular choice of factor- 
ization scale. The iy±jets and multijets samples have 
an overall uncertainty of 4% from their normalization to 
data This includes an uncertainty of 25% on the 
heavy flavor fraction of the M^±jets sample. 

The shape effects are modeled by shifting each source 
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TABLE IV: Input variables used in the neural network analysis. 



PT(jetl) 

PrOetlt^gg^d) 

r;(Iepton) 

PT{jetl,jet2) 
ffT(jetl,jet2) 
Pt(W) 

M(Vy,jetlt,gg,d) 
M(alljets) 
cos(jetl, lepton)iab 



Transverse momentum of the leading jet 
Transverse momentum of the f)-tagged jet 
Pseudorapidity [24] of the lepton 
Missing transverse energy 

Transverse momentum of the two leading jets 

Scalar sum of the transverse momenta of the two leading jets 

Transverse momentum of the reconstructed W boson 

Invariant mass of the reconstructed top quark using the W boson [25] and the fe-tagged jet 
Invariant mass of all jets 

Cosine of the angle between the leading jet and lepton in the laboratory frame of reference 



0.3 
0.25 

0.2 
0.15 

0.1 
0.05 



(a) 



■ w+jets 
tt 

SMtb 
SM tqb 
■FCNC 




100 



200 300 
H,(jet1,jet2) [GeV] 



0.14 
0.12 
0.1 
0.08 
0.06 
0.04 
0.02 




r (b) 


— W+jets 
.... 

■ SM tb 
SM tqb 

— FCNC 









0.5 1 
NN output 



♦D0 230pb^ 
-FCNC(xlO) 
SM tb+tqb 




NN output 



FIG. 2: Distributions of (a) an input variable to neural networks, and outputs normalized to (b) unity, and (c) 230 pb~^ of data 
with backgrounds summed. The FCNC signal is for the summed teg and tug processes, each evaluated at Hg/A — 0.03 TeV~^ 
(color online). 



of uncertainty by plus or minus one standard deviation 
with respect to its nominal value before any event selec- 
tions. The resulting uncertainties are: (i) (1-16)% due 
to jet energy scale, (ii) (2-8)% from trigger modehng, 
(iii) (1-5)% due to jet energy resolution, (iv) (1-9)% due 
to jet identification, and (v) (5-13)% from 6-tag model- 
ing. Since the iy-|-jets MC yield is normalized to data 



before 6-tag parametrization, we take into account the 
uncertainty from 6-tag modeling for this sample. 



We use a Bayesian approach to set upper limits [27 1 
on the FCNC coupling parameters. Given TV observed 
events, we define a Bayesian posterior probability density 
in a two-dimensional plane of (k^/A)^ and (k''/A)^ as: 



J 



p{[KyA]^K/Ar\N) (X 



LiN I n) p,ifcJu,b) P2(K/A]2) P3(K/A]2) d/,d/„d6, 



(1) 



r 



where L is a Poisson likelihood with mean n, and pi (i — 
1,2,3) are prior probability densities of the respective pa- 
rameters. The likelihood L is a product of the likelihoods 
over all bins of the neural network output distributions, 
n is the predicted number of events, equal to the sum of 
signal (s) and background (b) yields: 



^ s + b 

— Jc 



(2) 



(</A)^ + /„ X (</A)^ + b, 



where the constant factors fc and /„ are obtained from 
the simulated signal samples at Kg/A = 0.03 TeV~^. The 
prior probability density pi, is a multivariate Gaussian, 
with the mean and standard deviation defined by the 
estimated yields and their uncertainties, to take into ac- 



count correlations among the different samples and bins. 
Since the signal cross sections depend quadratically on 
Kg/A, for p2 and P3 we choose priors flat in (k^/A)^ and 
(Kg/A)^ respectively, which imply priors flat in the cor- 
responding cross sections. 

From the two-dimensional posterior probability den- 
sity, exclusion contours at different levels of confidence 
(fc) are defined as contours of equal probability that en- 
close a volume fc around the peak of the posterior density. 
These contours are shown in Fig.[3l using data from both 
electron and muon channels. The one-dimensional poste- 
rior probability density over any dimension is obtained by 
integrating the two-dimensional posterior over the other 
dimension. The resulting limits, translated to Kg/A, us- 
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ing data (observed limits) as well as the expected limits 
for which the observed count is set to the predicted back- 
ground yield in any bin, are summarized in Table |Vl 

To conclude, we analyzed 230 pb~^ of lepton+jets data 
collected at DO from pp collisions at a center of mass en- 
ergy of 1.96 TeV, and searched for presence of non-SM 
production of single top quarks. We found no deviation 
from SM predictions, and therefore set limits on anoma- 
lous coupling parameters, k^/A and k^/A, using multi- 
variate neural network discriminants. The 95% C.L. ob- 
served (expected) limits are 0.15 (0.16) TeV~^ on k^/A, 
and 0.037 (0.041) TeV"! on k^/A. These are first limits 
from hadron colliders on FCNC couplings of a gluon to 
the top quark and a charm or up quark, and a factor 
3-11 better than those from HERA. 



% 0.003 
I- 

0.002 
0.001 



D0 230 pB^ 



195% CL 
190% CL 
i68%CL 



r 



0.01 



0.02 0.03 0.04 
(Kj /A)' [TeV=] 

FIG. 3: Exclusion contours at various levels of confidence 
using 230 pb~^ of DO data in both the electron and muon 
channels (color online). 



TABLE V: Upper limits on k^/A and k^/A, at 95% C.L. 





Observed ( 


expected) limits [TeV ^] 




K^/A 


</a 


Electron channel 


0.16 (0.19) 


0.046 (0.052) 


Muon channel 


0.21 (0.21) 


0.049 (0.050) 


Combined 


0.15 (0.16) 


0.037 (0.041) 
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